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ABSTRACT 



models estimate 



The accuracy with which regression 
treatment effects is dependent upon a number of conditions. The 
stability cf the regression line <a function of sample size and 
TOrrelation between pretest and posttest) is said to be the most 
important cf these conditions. The utility of regression models is 
proportional to the size cf the correlation between pretest and 
posttest. AS the size of the correlation increases, the predicted 
posttest scores of the treatment group decreases. This Produces a 
corresccnding increase in the difference between predicted and 
observed scores. It is further stated that in compensatory education 
rrdects, factors which lower the correlation between pretest and 
posttest for low scoring students may invalidate the results. Given 
these condition, this study examined the impact cf the correlation 
be*wepn pretest and posttest on the accuracy «ith which regression 
models' *»sti mate treatment effects in Title I evaluation. More 
specifically, the effect of the correlation between pretest and 
ccsttest on the estimation of treatment affects with regression 
models was studied, usinq simulated data. Conclusions regarding the 
importance of the pretest-pc sttest correlation varied, depending on 
whether sore emphasis is placed upon unbiased estimates or 
efficiency. (Author/JKS) 
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PRETEST-POSTTEST CORRELATION AND REGRESSION MODELS 

Kim Onn Yap 
Northwest Regional Educational Laboratory 



INTRODUCTION 



The accuracy with which regression models estimate treatment effects 
is dependent upon a number of conditions. Of these conditions, the 
stability of the regression line (a function of sample size and 
correlatic- between pretest and posttest) is said to be the most 
important (Tallinadge s Wood, 1976; TalUnadge & Horst, 1976; Horst, 
Tallmadge & Wood, 1975) . 

Many writers (e.g., Horst, Tallmadge s Wood, 1975) have stated that 
the utility of regression models is proportional to the size of the 
correlation between pretest and posttest. As the size of the correlation 
increases, the predicted posttest scores of the treatment group 
decreases. This produces a corresponding increase in the difference 
between predicted and observed scores. It is further stated (Tallinadge . 
Horst, 1976) that in compensatory education projects, factors which lower 
the correlation between pretest and posttest for low scoring students may 
invalidate the results. 

Thus, according to these writers, in using regression models one 
should use a pretest which has a high correlation with the posttest. The 
higher the correlation, the lower the magnitude of regression to the mean. 
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on the other hand, it is also possible to argue that a high 
pretest-posttest correlation is not important. Since selection is based 
on the pretest, a high or low correlation should not affect our 
confidence in the no-treatment expectation. A high or low correlation 
merely reflects the goodness of the selection criteria. In the special 
case when the pretest-posttest correlation is zero, the regression models 
become equivalent to a control group design where students are randomly 
selected for treatment and control groups. 

While high correlations between pretest and posttest are recommended 
by proponents of regression models, the size of such correlations of 
course could not be pre-determined. In some cases (as when teacher 
ratings or some composite measure is used as pretest and a standardized 
test is used as posttest) the correlation can be expected to be low. 

The objective of the present study was to examine the impact of the 
correlation between pretest and posttest on the accuracy with which 
regression models estimate treatment effects in Title I evaluation. More 
specifically, the study was to provide an answer to the question: Does 
correlation between pretest and posttest make a difference in estimting 
treatment effects with regression models? 



PROCEDURE 



TO Study the impact of the size of pretest/posttest correlation on 
the accuracy with which regression models estimate treatment effects, 
data resembling those suited for analysis through these models were 
simulated. The rudiments of the simulation were as follows: 
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^lij = Xij + Eij 

Yjij = Xij + Gij + TEij + E'ij, (2) 
where Y^^ij is the pretest score of student i in group j; Yzij is the 
posttest score of student i in group j; Xij is the true achievement level 
of student i in group j at pretest; Gij is growth attributable to factors 
other than the treatment for student i in group j; TEij is the treatment 
effect for student i in group j; and Eij and E'ij are error terms. 

For purposes of the simulation, it was assumed that the mean growth 
rates (Gij's) for the treatment and control groups are equal. In 
equation (2), TEij's were set to equal zero for students in the control 
group to indicate the absence of treatment effects. 

The values of Xij, Gij, TEij, Eij, and E'ij were made up of random 
numbers provided by GAUSS (IBM, 1968), a computer subroutine which 
generates normally distributed random numbers. The relative size of Xij, 
Eij, and E'ij was adjusted by means of multipliers. For example, the 
values Of a set of Xij, Eij and E'ij may be obtained as follows: 
Xij = .7 (Nj^) 
Eij = .3 (N2) 
E'ij » .3 (N^) 

where the Ns are random numbers. Means and standard deviations for the 
random numbers were chosen in such a way that Y^^^j yjij would 
have approximately a mean of 50 and a standard deviation of 21.06, 
respectively, to correspond with the mean and standard deviation of 
Normal Curve Equivalents (NCEs) . For example, in 




3 2/79 
NWREL TAC _ 3372A 

O 



^lij * Xij + Eij, where 
Xi j = . 7 (Nj^) , and 

Eij = .3 (N^), 

both Nj^s and N2S were given a mean of 50 and a standard deviation of 
27.65. This gave Y^ij a mean of 50 and a standard deviation of 

J (.7)2 (27.65)2 + (.3)2 (27.65)2 

which simplifies to 21.06. 

The same procedure was used to give Y2ij a mean of 50 and a 
standard deviation of 21.06. Means and standard deviations for Gij and 
TEij were determined by providing the appropriate parameter values to 
subroutine GAUSS. Gij was set to have a mean of 10 and a standard 
deviation of 10 and TEij was set to have a mean of 7 and a standard 
deviation of 7. These means and standard deviations had been chosen to 
reflect what is most likely to occur in real-life situations in terms of 
NCE sr'.es. 

Negative values provided by GAUSS, which occurred on few occasions, 
were dropped, resulting in slightly higher means and lower standard 
deviations for the variables. 

The pretest data (Y^^ij) were first simulated. The hypothetical 
cases in each data set were rank-ordered. A strict cut-off, located at 
the 25th percentile point, was used to assign cases to the treatment 
(j Pi 1) and control (j = 2) groups. 

After the hypothetical cases had been assigned to Title I or control 
groups, posttest data (Yjij) were simulated by means of equation (2), 
adding growth (Gij) and treatnient effects (TEij) to pretest scores of 
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students in treatment groups and only growth (Gij) to pretest scores of 
students in control groups. 

Three critical parameters were manipulated in the simulation (see 
Table 1). First, pretest reliability was varied from .84 to .69. 
Second, the size of correlation between pretest and posttest was varied 
from .75 to .50 and then to .25. Third, sample size was made to vary 
from 100 to 200. 



Table 1 about here 



The manipulation of data reliability was based on Gulliksen's (1950) 
idea that a reliability coefficient can be expressed as the ratio of true 
variance to total variance. This means that we could vary reliability by 
applying different multipliers to the random numbers which make up the 
values of variables. For example, in 

Yj^ij s xij + Eij, where 

Xij = .7 (Nj^), 

Eij = .3 (N2), 

and Nj^s and N2S are given the same variance, the reliability 
coefficient of "^-^ij is given by 

ryiyi - =r^- 

•^•K var Xij + Var Ei] 

Since multiplying a set of numbers by a constant increases the 
variance by the square of the constant and since Nj^s and N2S have th« 
same variance, we have 



= .84. 
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That is, the reliability of Y^^ij is .84. It could readily be 
verified that by changing the multipliers to .6 for N^^g and .4 for 
Njs we will have lowered data reliability to .69. In the simulation, 
data sets with pretest reliability of .69 and .84 were created. 

Correlations between Y^^j and Y2ij were controlled by means of 
the following formula: 

yf^ll rii 

Reported by Gulliksen (1950, p. 101), the formula gives the 
correlation between a test and a criterion when each is increased to 
infinite length to attain a reliability of unity. Given that 

■^^lij = Xij + Eij, and 

the two variables share a common true score component in Xi j , with R^p- 
reaching unity when both Y^^^j and Y2ij are made perfectly reliable. 
(For simulation purposes, variance given to Gij was treated as error 
variance.) It follows that ^r^^ = rn, which provides a means 
of obtaining a desired value for r^^^ by changing r^, r^, or both. 

In the present simulation we had required that r^^ (reliability of 
^lij) be either .84 or .69 (a fixed value), leaving rn (reliability 
of Yjij) to be varied to yield a desired value for rn. The way in 
which a desired correlation, say .75, between Y^^ij and Y2ij 
obtained is illustrated as follows: 



was 



Since (a) ^^/"rnT^ii" = rn, 
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(b) the desired value of r^^j „as .75 and (c) rn had been given a 
reliability of .84, we had 



84 rj^ = .75 

^ " /.S4 

= .82 
rii = .67 



In Other words, giving Y^^^ ^ reliability of .67 produced a 
cxarrelation of .75 between Y^^j^j g^d Y2ij. 

Since the variance of Y2ij „as made to equal 443.52 (the square of 
21.06), the true variance required to yield a reliability coefficient of 
.67 was (443.52))t(.67) which equals 297.16. An appropriate multiplier 
(.62 in this case) was then applied to Xij in Y2ij (xij had a standard 
deviation of 27.65 when pretest reliability was .84) to produce the 
required true variance. 

Ito Obtain correlation coefficients of .75, .50 and .25 between Y^ij 

and Y2ij, the reliability of Y2ij was varied from .67 to .07 fOr data 
sets where pretest (Yj^ij) reliability was .84 and from .78 to .09 for 
data sets where pretest reliability was .69. However, as indicated 
earlier, variance due to Gij was treated as error variance. Thus, the 
"reliability" coefficient for Y2ij „as more a measure of the amount of 
cxsvariance between Y^^ij and Y2ij than the ratio between true and 
total variance in Y2ij itself. When variance due to Gij was treated as 
part of the true variance, the reliability of Y2ij y/as found to range 
from .30 to .90 for data sets where pretest reliability was .84 and fran 
.32 to .99 for data sets where pretest reliability was .69. 
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The correlations between ^^i-j and Y2ij were determined before 
treatment effect {TEij) was added as a component to Y^i^ for Title I 
groups. Since TEij was given a population standard deviation of 7 (or a 
variance of 49) , the correlations between Y^^^j and Y2ij for Title I 
groups would differ slightly from the correlations for the comparison 



groups. 



THE DATA SETS 



Taking into account the different levels of each of the three 
parameters (i.e., size of correlation between pretest and posttest, data 
reliability, sample size), a total of 3 x 2 x 2 (or 12) categories of 
data sets were simulated. One hundred data sets were created for each of 
the categories. Characteristics of these data sets are summarized in 

Appendices A to L. 

An examination of the characteristics of these data sets suggests 
that they closely resemble what we had intended to create. The obtained 
values, in some instances, deviate slightly from the parameters. As 
explained earlier, this came about essentially as a result of dropping 
negative values provided by GAUSS on a few occasions. Except for the 
slightly higher means and lower standard deviations, the data have the 
appearance of NCE scores. (The higher means for Y^^^ are due to higher 
means for Xij and Gij.) 
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ANALYSIS AND RESULTS 



Data analysis procedures described by Tallmadge and Wood (1976, 1978) 
for regression models we.e applied to the simulated data. Essentially, 
the procedures require that a regression line be determined on the basis 
of comparison group data to predict what the performance of Title I group 
would have been-if they had not received Title I treatment. The 
prediction is made at the point where the Title I group's pretest mean 
intercepts the regression line. The predicted performance is then 
subtracted from the actual performance of the Title I group, with the 
remainder being the estimated treatment effect or gain. 

in the simulation this ^estimated gain was again subtracted from the 
actual gain (TEij) which was built into the posttest (Yjij) of the 
Title I group. The difference was then interpreted as an index of the 
accuracy with which regression models estimate treatment effects. The 
means and standard deviations of such differences for each of the 12 data 
categories are summarized in Table 2. 



Table 2 about here 



DISCUSSION 



Before we examine the effects which the manipulated parameters (i.e., 
pretest-posttest correlation, pretest reliability, sample size) have on 
the estimation of treatment effects it might be helpful to present a 
perspective in which the results will be interpreted. As Wonnacott and 
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Wonnacott <1970) point out, an estimator may be described in terit»s of 
bias ef^i°ien<=y and consistency. An unbiased estimator is one that i^' 
on the right on target. In other words, its expected vai^e is 

identical "^^^^ the true value of the parameter. A biased estimator, oO 
the other hand, has an expected value that is "off target" or deviates 
from the ^^^^ ^lu^ of the parameter. An efficient estimator is an 
unbiased estimator with a relatively small variance. An inefficient 
estimator' °" the other hand, is an unbiased estimator with a restively 
large vaJ:^^"=e. a consistent estimator is one which zeroes in on the 
true vaiu^ °^ the parameter as sample size increases. 

^ie^e^ in this perspective, the results in Table 2 suggest that 
regression models can be depended upon to provide relatively unbiased 
estimates °f treatment effects. This is so because the mean differences 
between estimated and actual gains were in general negligibly small- 
only in ^ne case (Category V) did the mean difference exceed an absolute 
value of ^-O- It is interesting to note that none of the parameters 
being „^iPulated in the simulation appeared to have any appreciable ot 
systematic effects on the amount of bias in estimation. 

gowe^^"' with respect to efficiency we have a different story. The 
results that the size of correlation between pretest and posttest 

has a cl^^^ly discernible effect on the efficiency of the estimates- 
This is "Reflected in the increasingly larger standard deviations for 
mean di£^^""ces as we move from a higher correlation to a lower 
cortelati°"* This pattern is seen across all 12 categories of data 
sets. ^"""^ ^^^^^ ^^"^^^ categories (where the pretest has a 
reliability of .84) the standard deviation of the mean differences 
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increases from 3.53 to 4.93 as the pretest-posttest correlation decreases 
froih .78 to .25. For the next three categories (where the pretest 
reliability is lowered to .69) the increase in standard deviation is even 
greater, rising from 4.47 to 6.18 as the pretest-posttest correlation 
drops from .75 to .27. Similar patterns are seen in data sets in the 
other categories. 

Sample size also seems to have some effect on the distribution of the 
mean differences or the efficiency of the estimates. This is evidenced 
by the fact that standard deviations for data categories VII to XII 
(where sample size is 200) generally are smaller than those for the first 
six categories (where sample size is 100). The former standard 
devi; 'ions range from 2.88 to 4.59, the latter from 3.53 to 6.18. It 
would appear that a smaller sample size serves to further reduce the 
efficiency of the estimates. 

That regression models do not provide us with efficient estimates of 
treatment effects is perhaps more dramatically shown by the ranges of the 
mean differences obtained for the various data categories. These are 
reported in Table 3. As one can hardly fail to be struck by the 
magnitude of the ranges, one must remember that these are the extremes. 
While they could occur and did occur in the simulation, the probability 
of their occurring in any single evaluation is low. The standard 
deviations of mean differences discussed earlier remain a more stable 
measure of the efficiency of the estimates. 

Table 3 about here 
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AS indicated earlier, an increase in sample size appears to enhance 
the efficiency of the estimates. A larger sample size, however, does not 
appear to produce a more consistent estimator-at least not in any 
systematic way. It is true that in absolute value the sum of mean 
differences obtained for data categories VII to XII (where sample size is 
200) is less than that obtained for data categories I to VI (where sample 
size is 100), the former being 1.75 and the latter 2.2. It is also true, 
however, that in absolute value the mean differences for data categories 
I and II are smaller than the mean differences for data categories VII 
and VIII. This is also true with respect to data categories IV and VI 
and da-a categories X and XII. It would appear that with regression 
models a larger sample size does not necessarily result in an increase in 
the consistency of estimates. 

CXJNCLUDING REMARKS 

in the simulation the author had set out to obtain evidence which 
would support one of the two arguments concerning the importance of 
having a high pretest-posttest correlation when regression models are 
used to estimate treatment effects. Quite ironically, the evidence, as 
it turned out, appeared to support both arguments in different ways. If 
one used bias as the sole criterion to judge the adequacy of an 
estlisator, the evidence would sugge^^t that the size of pretest-posttest 
^crelation appears to have neither appreciable nor systematic effects on 
t*ie rstlmation of treatment effects. In general, regression models 
provide estimates that are practically unbiased. 
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This does not mean, however, that pcetest-posttest correlation makes 
no difference whatsoever. For if one used efficiency as a criterion to 
assess the adequacy of an estimator, the evidence would suggest that 
pretest-posttest correlation has appreciable and systematic effects on 
the estimation of treatment effects. The pattern of results clearly 
indicates that the lower the correlation the less efficient the 
estimates. In addition, sample size also appears to have some effect on 
efficiency, with larger sample sizes tending to enhance the efficiency of 
the estimates. 

Therefore, in the case of a single evaluation-which is typically the 
case in a Title I di.trict-the effects of pretest-posttest correlation 
on evaluation results can by no means be ignored. It would not be 
difficult to see why upon a perusal of the ranges of mean differences 
reported in Table 3 a Title I project administrator would seek a higher 
pretest-posttest correlation or a different evaluation model to estimate 

treatment effects. 

Perhaps the most significant findings of the present study relate to 
the overall adequacy of regression models when used to estimate Title I 
treatment effects. In a typical situation, the pretest- 
posttest correlation derived from Title I evaluation data would fall 
within the range of those simulated in the present study (i.e., from .25 
to .75). The results of the simulation suggest that within that range of 
correlation coefficients, regression models provide relatively 
inefficient-although relatively unbiased-estimates of treatment 
effects. The confidence intervals (as indicated by the standard 
deviations of mean differences) are clearly too large to be sensitive to 
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s„,aU achievement gains typically produced in Title I projects. It would 
appear that the correlation coefficient of .40 reconunended by Tallmadge 
and wood (1978, p. ID as, by rule of thumb, being sufficiently high to 
ensure reasonable accuracy of estimates of treatment effects, may, after 
all, be too liberal. The results of the present study would suggest a 
correlation coefficient of .75 or higher. 
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Table 1 

Parameters manipulated in the Simulation Study 



Parameter 
Size of Pretest-Posttest 


Level 
.75, .50, .25 


Correlation 




Pretest Reliability 


.84, .69 


Sample Size 


100, 200 
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Table 2 



Differences Between Actual and Estimated Ga ins by D 
Category and Size of Pretest -^Posttest Correlati 



Data Set 
Category 


Pretest-Posttest 
Correlation 


Estimated 
Gain 


Actual 
Gain 


Differ 

CIVZ Cll 1 


ence 
S.D. 


I 


.78 


8. 82 


8. 


89 


• u o 


3 53 


II 


.54 


9.15 


8. 


86 


90 


A 88 


III 


.25 


8.45 


8. 


93 


.48 


4.93 


IV 


.75 


8.81 


8. 


89 


.09 


4.47 


V 


.53 


9.91 


8. 


86 


-1.05 


5.28 


VI 


.27 


9.14 


8. 


93 


-.21 


6.18 


VII 


.78 


9.07 


8. 


88 


-.19 


2.88 


VIII 


.54 


8.69 


9. 


05 


.36 


3.83 


IX 


.24 


8.81 


8. 


97 


.16 


4.78 


X 


.75 


9.06 


8. 


88 


-.18 


2.69 


XI 


.52 


9.01 


9. 


05 


.04 


4.07 


XII 


.26 


8.14 


8. 


97 


.82 


4.59 
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Table 3 



Range of Di fferences Between Act ual and Estimated Gains 
b y DiirsiTcateaorv and Sir -e of Pretest-Postte_s_^orrelation 



Data Set 
Category 


Pretest-Posttest 
Correlation 


Prom 


f f erences 
To 


I 


.78 


-7.51 


8.02 


II 


.54 


-12.24 


13.37 


III 


.25 


-8.63 


14.86 


IV 


.75 


-9.38 


9 . 33 


V 


.53 


-16. as' 


13.4? 


VI 


.27 


-13.48 


12.37 


VII 


.78 


-6.95 


6.53 


VIII 


.54 


-7.64 


9.59 


DC 


.24 


-12.66 


13.04 


X 


.75 


-6.63 


6.46 


XI 


.52 


-10.87 


8.69 


XII 


.26 


-12.46 


13.62 
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Footnotes for Appendices A-L 



1. The notations in the Appendices are interpreted as follows; 







pretest reliability 


'^yiy2 


a 


correlation between pretest and posttest 


^1 




pretest mean 


syi 




pretest standard deviation 


^2 


s 


posttest mean 


Sy2 




posttest standard deviation 


6 




growth mean 


sg 


a 


growth standard deviation 



Each data category consists of 100 data sets. For categories I-VI, 
each of the 100 data sets consists of 100 simulated cases. For 
categories VII-XII, each of the 100 data sets consists of 200 
simulated cases . 

S.D. in the last column refers to standard deviations for the 100 
simulated data sets. 
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Appendix A 
Characteristics of Data Sets in Category 



Characteristics 


Mean 


S.D. 




.84 






.78 


.04 


^1 


52.33 


1.73 




19.38 


1.19 


Y2 


55.76 


1.75 




18.43 


1.17 


6 


12. 91 


.83 


Sg 


7.91 


,55 
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Appendix B 

Characteristics of Data Sets in Category 



Characteristics 


Mean 


S.D. 




.84 






.54 


.08 


^1 


52.10 


1.95 


syi 


19.42 


1.30 




56 . 08 


1.76 


sy2 


17.89 


1.17 


G 


12.86 


.75 


sg 


7.96 


.57 
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Appendix C 

Characteristics of Data Sets in Category 



Characteristics 




S.D. 




.84 






.25 


.09 


^1 


52.42 


1.75 


syi 


19.53 


1.27 




54.69 


1.76 


Sy2 


18.28 


1.28 


G 


12.78 


.89 


sg 


7.83 


.53 
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Appendix D 
rharaeteristics Of Da<-a Sets in Category 



Characteristics 


Mean 


S. D. 
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Appendix E 

Ch;^raeteristics of Data Sets in Category V 



CharaC^^e'^istics 



^2 
(5 



Mean 
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Appendix F 
C!haracteristics of Data Sets in Category 



Characteristics 


Mean 


S.D. 
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Appendix G 

rharaeteristics of Data Sets in Category VU 




ryiy2 
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^2 

Sy2 
G 

sg 



.84 
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52.16 
19.35 
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Appendix H 

Characteristics of Dat a Sets in Category VIII 



Characteristics 

ryiy2 
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^2 
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G 
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.84 
.54 
51.86 
19.29 
56.42 

18.19 
12.96 

7.94 



S.D. 



.05 
1.30 

.98 
1.40 

.84 
.56 
.39 
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Appendix I 
rh.:>raeteristics of Data Sets in Category 



Character istics 


Mean 


S.D. 
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Appendix J 

rharafiteristics of Data Sets in Category X 
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Appendix K 

rharaefceristics of Data Sets in Category 



Characteristics 


Mean 


S.D. 
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Appendix L 
Pharaeteristics of Data Sets in Category 



Characteristics 


Mean 
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